Modular networking hardware platform

ABSTRACT

A modular networking hardware platform utilizes a combination of different types of units that are pluggable into cassette endpoints. The present disclosure enables the construction of an extremely large system, e.g., 500 Tb/s+, as well as small, standalone systems using the same hardware units. This provides flexibility to build different systems with different slot pitches. The hardware platform includes various numbers of stackable units that mate with a cost-effective, hybrid Printed Circuit Board (PCB)/Twinax backplane, that is orthogonally oriented relative to the stackable units. In an embodiment, the hardware platform supports a range of 14.4 Tb/s-800 Tb/s+ in one or more 19″ racks, providing full features Layer 3 to Layer 0 support, i.e., protocol support for both a transit core router and full feature edge router including Layer 2/Layer 3 Virtual Private Networks (VPNs), Dense Wave Division Multiplexed (DWDM) optics, and the like.

FIELD OF THE DISCLOSURE

The present disclosure generally relates to networking hardware. Moreparticularly, the present disclosure relates to a modular networkinghardware platform.

BACKGROUND OF THE DISCLOSURE

Networks, data centers, cloud infrastructure, etc. are realizedultimately through physical hardware. The physical hardware is used torealize networking, computing, and/or storage functions, includingoptical/photonic networking, Time Division Multiplexing (TDM), packetswitching, etc. Conventionally, there are generally two approaches forhow physical hardware is deployed—through a chassis and associatedmodules or through a rack mounted unit. A chassis-based system includesa chassis with multiple slots and a backplane that support differentmodules, line cards, circuit packs, blades, etc. (for simplicity, theseare referred to as modules). The chassis-based system allows deploymentflexibility where modules as-needed or required are selectively insertedin the chassis. The chassis includes an electrical (Printed CircuitBoard (PCB)) backplane, fixed-size modules, shared power and cooling forall of the modules, a central switch fabric such as through one or moremodules, etc. Other advantages of a chassis include power installationonce for the chassis, front access only, redundancy, tight coupling ofsoftware and hardware, etc.

There are several fundamental challenges, namely 1) PCB backplanes arebecoming limited in their ability to support higherSerializing-Deserializer (SERDES) rates at the same Fabric-to-Moduledistances, 2) the industry business model that led to these types ofdesigns is changing, 3) consumption patterns on life-cycle anddepreciation practiced by network operators, that is in turn reinforcedby the merchant Application Specific Integrated Circuit (ASIC) supplychain, connote requirements for greater modularity, and the like. Thatis, the conventional chassis-based approach is centralized with a fixedform-factor, which limits flexibility in size, power, and the ability toaddress different applications. Further, the chassis-based approachrequires a lock-in ahead of time, suboptimal faceplate use, fixed slotsizes, and power, fixed cooling, fixed slot bandwidth, etc. Thechassis-based approach requires a large initial investment.

The rack mounted unit is also referred to as “pizza boxes” and generallyincludes 1-2 (or more) Rack Unit (RU or simply U and which is a standardunit of measurement of 44.5 mm or 1.75″) high completely enclosed formfactor. The rack mounted unit differs from a chassis in that all of thehardware is already included as the rack mounted unit is deployed. Forexample, data centers realize network elements such as in a leaf/spinearchitecture in a completely decentralized and modular form using rackmounted units. Advantages of the rack mounted unit approach includestandard interfaces, stackability, low cost, adaptability, low first-incost, and pay as you go, etc. While this provides flexibility, thisapproach has interconnect complexity, such as massive amounts of cablingin the front and this cabling is based on pluggable optics that lack thedensity achievable in the chassis-based approach. Also, this approachdoes not include redundancy, expansion to larger systems is complex andinefficient, and each install requires a power connection.

An example solution to the above-referenced limitations is described incommonly assigned U.S. patent application Ser. No. 15/959,746, filedApr. 23, 2018, and entitled “Modular network element architecture,” thecontents of which are incorporated herein by reference.

BRIEF SUMMARY OF THE DISCLOSURE

The present disclosure relates to a modular networking hardwareplatform. The hardware platform utilizes a combination of differenttypes of units that are pluggable into cassette endpoints. An aspect ofthe present disclosure includes the construction of an extremely largesystem, e.g., 500 Tb/s+, as well as small, standalone systems using thesame hardware units. This provides flexibility to build differentsystems with different slot pitches. The hardware platform includesvarious numbers of stackable units that mate with a cost-effective,hybrid PCB/Twinax backplane, that is orthogonally oriented relative tothe stackable units. In an embodiment, the hardware platform supports arange of 14.4 Tb/s-800 Tb/s+ in one or more 19″ racks, providing fullfeatures Layer 3 to Layer 0 support, i.e., protocol support for both atransit core router and full feature edge router including Layer 2/Layer3 Virtual Private Networks (VPNs), Dense Wave Division Multiplexed(DWDM) optics, and the like.

In an embodiment, a networking hardware platform includes a plurality ofunits each having a first plurality of connectors on a rear side,wherein each of the plurality of units is any of a packet unit, a fabricunit, and a control unit, and wherein collectively the first pluralityof units are configured to operate as a network element in a network;and a cassette system including a one or more units each having cablingtherein and a second plurality of connectors located along the length toconnect to corresponding connectors of the first plurality ofconnectors, wherein the cassette system is configured to connect to arack, and wherein an orientation of the one or more units is orthogonalto an orientation of the plurality of units, in the rack, wherein thecabling, in each of the one or more units, extends between each of thesecond plurality of connectors for interconnectivity with correspondingconnector along the length of the corresponding one or more units. Thecabling can include a plurality of Twinaxial cables. The secondplurality of connectors can be further configured to connect tocorresponding connectors in a second rack, such that the network elementincludes the plurality of units in the rack and in the second rack. Thenetworking hardware platform can further include a second control unitlocated in the second rack connected to a first control unit located inthe rack.

The cabling can include optical cabling with a multi-fiber connectorforming the first plurality of connectors and the second plurality ofconnectors. The networking hardware platform can further include one ormore fans disposed at an opposite end of the one or more units from theplurality of units. The one or more units can include multiple unitsthat form channels for airflow between the plurality of units and thefans and for electromagnetic interference (EMI) containment. Each of theplurality of units can be in a different plane from one another andairflow is front-to-back from a faceplate of each unit to correspondingone or more fans. The one or more units can be each a passive device andincludes a read-only memory for identification thereof. A depth of theone or more units can be based on how much interconnect is required forthe plurality of units. A pitch of at least two units of the pluralityof units can be different. The packet units can be configured to supportpluggable optical modules. The networking hardware platform can furtherinclude at least one universal unit that is similar functionally to thepacket unit except for larger pitch and for support of at least onecoherent optical module for long haul transmission. The networkinghardware platform can further include N+1 power modules located at arear of the one or more units, N is an integer ≥1 and the +1 is aprotection module. The N+1 power modules can provide substantially 380VDC to the units.

In another embodiment, a method of installing a networking hardwareplatform includes installing a cassette system in a rack, wherein thecassette system includes one or more units each having cabling thereinand a first plurality of connectors located along the length, whereinthe cabling, in each unit of the cassette system, extends between eachof the first plurality of connectors for interconnectivity between eachcorresponding connector along the length of the corresponding unit ofthe cassette system; and connecting a plurality of units to the cassettesystem, wherein each of the plurality of units has a second plurality ofconnectors on a rear side configured to connect to correspondingconnectors of the first plurality of connectors, wherein an orientationof the plurality of units is orthogonal to an orientation of the one ormore units, wherein each of the plurality of units is any of a packetunit, a fabric unit, and a control unit, and wherein collectively theplurality of units are configured to operate as a network element in anetwork. The cabling can include a plurality of Twinaxial cables. Themethod can further include connecting the first plurality of connectorsassociated with the cassette system to corresponding connectors in asecond rack, such that the network element includes the plurality ofunits in the rack and in the second rack.

In a further embodiment, a networking hardware interconnection systemincludes one or more units that are configured to connect to a rack; andcabling in each of the one or more units between a first plurality ofconnectors located along the length, wherein the first plurality ofconnectors are configured to connect to corresponding connectors of asecond plurality of connectors of a plurality of units that areorientated orthogonal to the one or more units, and wherein the one ormore units form channels for airflow between the plurality of units andfans and for electromagnetic interference (EMI) containment. The cablingcan include a plurality of Twinaxial cables.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated and described herein withreference to the various drawings, in which like reference numbers areused to denote like system components/method steps, as appropriate, andin which:

FIGS. 1A, 1B, and 1C are block diagrams of a networking hardwareplatform with FIG. 1A illustrating a rear view of a rack with cassettes,FIG. 1B illustrating a front view of stackable horizontal units, andFIG. 1C illustrating modules that plug into either a front or a rear ofthe units;

FIG. 2 is a block diagram of a front view of an example hardwareplatform supporting 230 Tb/s in a single rack;

FIGS. 3A-3D are diagrams of a single rack hardware platform 10 whereFIG. 3A is a side view, FIG. 3B is a front view, FIG. 3C is a rearsection view, and FIG. 3D is a rear view;

FIG. 3E is a block diagram of a hardware platform in a side view tocompare a cassette backplane with a backplane-less embodiment;

FIG. 4 is a block diagram of a front view of single rack illustratinghardware platforms of varying sizes;

FIG. 5 is a block diagram of a front view of three racks illustrating amulti-rack configuration for a hardware platform;

FIGS. 6A-6D are block diagrams of an ortho-direct approach for achassis, for comparison with the hardware platform;

FIGS. 7A and 7B are block diagrams of an ortho-direct approach for achassis with fabric module couplers for expansion;

FIG. 8 is a table illustrating different approaches for networkinghardware to compare the hardware platform of the present disclosure withthe chassis of FIGS. 6-7 and pizza boxes;

FIG. 9 is a block diagram of a packet unit, a server unit, and a controlunit for describing a control plane and OAM plane scalability;

FIG. 10 is a diagram illustrating a leaf-spine architecture;

FIG. 11 is a diagram comparing the hardware platform to the leaf-spinearchitecture;

FIG. 12 is a diagram of a hardware platform illustrating multi-rackexpansion;

FIG. 13 is a diagram of a rear view and a side view of the cassettes;

FIG. 14 is a side view of the hardware platform 10 illustrating airflowbetween the units, the cassettes, and the fans;

FIG. 15 is a perspective diagram of an implementation of the cassettesand housings for the packet units and the fabric units;

FIG. 16 is a front perspective diagram of the housings connected to thecassettes and mounted in a rack;

FIG. 17 is a rear perspective diagram of the same hardware platform asFIG. 16;

FIG. 18 is a horizontal section view of the same hardware platform asFIG. 17;

FIG. 19 is a rear view diagram of the housings illustrating theconnectors to the cassettes; and

FIG. 20 is a flowchart of a process of installing a networking hardwareplatform.

DETAILED DESCRIPTION OF THE DISCLOSURE

Again, the present disclosure relates to a modular networking hardwareplatform. The hardware platform utilizes a combination of differenttypes of units that are pluggable into cassette endpoints. An aspect ofthe present disclosure includes the construction of an extremely largesystem, e.g., 500 Tb/s+, as well as small, standalone systems using thesame hardware units. This provides flexibility to build differentsystems with different slot pitches. The hardware platform includesvarious numbers of stackable units that mate with a cost-effective,hybrid PCB/Twinax backplane, that is orthogonally oriented relative tothe stackable units. In an embodiment, the hardware platform supports arange of 14.4 Tb/s-800 Tb/s+ in one or more 19″ racks, providing fullfeatures Layer 3 to Layer 0 support, i.e., protocol support for both atransit core router and full feature edge router including Layer 2/Layer3 Virtual Private Networks (VPNs), Dense Wave Division Multiplexed(DWDM) optics, and the like.

Variously, the hardware platform includes the stackable units that matewith cassette endpoints.

The following acronyms are utilized herein

Acronym Phrase AOC Active Optical Cable BFD Bidirectional ForwardingDetection BGP Border Gateway Protocol CE Customer Edge CP Control PlaneCPU Central Processing Unit FEC Forwarding Equivalence Class FDBForwarding Database FIB Forwarding Information Base GE Gigabit EthernetICMP Internet Control Message Protocol IGP Interior Gateway Protocol IPInternet Protocol ISIS Intermediate System-Intermediate System LDP LabelDistribution Protocol MAC Medium Access Control MPLS Multi-ProtocolLabel Switching MSE Multi-Service Edge NGROADM Next Generation ROADM NICNetwork Interface Card NNI Network-Network Interface OAM Operations,Administration and Maintenance OSS Operations Support System OTN OpticalTransport Network PCE Path Computation Element PCEP PCE Protocol PEProvider Edge PIM Protocol Independent Multicast PIP Private IP PSPacket Switching PW Pseudo Wire QoS Quality of Service QSFP Quad SmallForm factor Pluggable RAM Random Access Memory RI Router Instance RIBRouting Information Base ROADM Reconfigurable Add Drop Multiplexer RSVPResource reSerVation Protocol SDN Software-Defined Network SES SwitchedEthernet Service TDM Time Division Multiplexing TE Traffic EngineeringUNI User-Network Interface VPN Virtual Private Network

Platform and Components

FIGS. 1A, 1B, and 1C are block diagrams of a networking hardwareplatform 10 with FIG. 1A illustrating a rear view of a rack withcassettes 12, FIG. 1B illustrating a front view of stackable horizontalunits 14, 16, 18, 20, 22, and FIG. 1C illustrating modules 24-34 thatplug into either a front or a rear of the units 14, 16, 18, 20, 22. Thestackable horizontal units 14, 16, 18, 20, 22 include a packet unit 14,a universal unit 16, a fabric unit 18, a control unit 20, and a serverunit 22. Each of the stackable horizontal units 14, 16, 18, 20, 22 canbe a rack-mounted unit with connectors (not shown in FIG. 1) for matingwith the cassettes 12. Those of ordinary skill in the art are familiarwith chassis and associated modules as well as with rack-mounted units(“pizza boxes”). The stackable horizontal units 14, 16, 18, 20, 22 are ahybrid implementation relative to the chassis with modules and thestand-alone pizza boxes. Further, those of ordinary skill in the artwill appreciate the horizontal units 14, 16, 18, 20, 22 are presentedfor illustration purposes, and other types of units are alsocontemplated and expected due to the modular nature of the hardwareplatform 10. Also, as described herein, element 12 is referred to as acassette 12 and one or more cassettes 12 can be referred to as acassette system. Further, the cassette system can include one or moreunits, each unit being a single cassette 12. Such a unit includes ahousing, cabling, and connectors.

Of note, the terms “horizontal” and “vertical” may be used herein forillustration purposes. The stackable horizontal units 14, 16, 18, 20, 22are shown as horizontally oriented whereas the cassettes are shown asvertically orientated. Those of ordinary skill in the art willappreciate the stackable horizontal units 14, 16, 18, 20, 22 could bevertically oriented whereas the cassettes 12 could then be horizontallyoriented. That is, the orientation of the cassettes 12 and theorientation of the units 14, 16, 18, 20, 22 are orthogonal to oneanother, and the reference to stackable horizontal units 14, 16, 18, 20,22 is merely for illustration of an example embodiment.

The modules 24-34 include includes a power module 24, a Universal SubSlot Module (US SM) 26, a switch module 28, a control and timing module30, an access panel module 32, and a fan module 34. The power module 24can provide power connectivity. The USSM 26 can be a standardizedpluggable interface for various types of pluggable modules includecoherent optics modules and the like. The switch module 28 can be apacket switch. The control and timing module 30 can provide Operations,Administration, and Maintenance (OAM) functions. The access panel module32 can provide management interfaces (e.g., Ethernet, etc.). The fanmodule 34 can provide cooling.

The packet unit 14 can include various ports, such as UNI/NNI400GE/200GE/100GE interface ports. In an embodiment, the packet unit 14can be 1 RU, including 36×QSFP4DD modules on the faceplate (14.4 Tb/scapacity and processing circuitry (x86+FPGA accelerator) for localcontrol and OAM functions. The packet units 14 can perform packetprocessing functions such as, for example, FIB/FDB operations,queuing/policing, access-control filtering/lists, control plane offloadfunctions (e.g., telemetry/OAM/BFD/etc.). The packet units 14 caninclude packet processing circuitry.

The universal unit 16 can have the same packet processing circuitry asthe packet unit 14 but support the USSM 26 to add coherent opticalinterfaces and/or unique capabilities such as circuit emulation ofOTN/Ethernet over a packet fabric/network. The USSM 26 is a pluggablemodule, specified by Ciena Corporation, and described, for example, incommonly assigned U.S. patent application Ser. No. 16/391,642, filedApr. 23, 2019, and entitled “Universal sub slot architecture fornetworking modules,” the contents of which are incorporated herein byreference. For example, the universal unit 16 has a larger footprint (3RU) relative to the packet unit 14 (1 RU) and can support up to 12 USSMs26. The universal unit 16 still utilizes the same forwarding chipsets asthe packet unit 14. In order to maximize the forwarding capabilities ofthe forwarding plane, the pitch of the universal unit 16 will be 3 RU asDWDM line interfaces cannot be packed as densely within an assembly asregular client optics can, in the packet unit 14.

The USSMs 26 allow for more interface and service flexibility, relativeto standard pluggable modules such as QSFP4DD. For example, the USSM 26can be an OTN/Ethernet module for circuit emulation of OTN/Ethernet overa packet fabric/network with 8×QSFP28 (of which 2 ports are QSFP-DD),with support for Flexible Ethernet (FlexE) if needed. The USSM 26 canalso be an integrated DWDM line interface, such as 2×200-800 Gb/smodems, including 600 G ULH, 4×CFP2-DCO modems (4×400 G ZR++ capable),etc.

The fabric unit 18 provides interconnect between packet units 14 anduniversal units 16. In an embodiment, the fabric unit 18 is a 1.25 RUsize and includes two switch modules 28.

The control unit 20 is a network element controller and can be based onan x86. The control unit 20 performs all aspects of system control planefunctions, including (but not limited to) Configuration Managementfunctions, Northbound (OSS) interfaces (Alarms, Telemetry,Provisioning), Routing Protocol control plane functions, and the like.

The server unit 22 is capable of performing any general processing forvarious use cases and can connect directly to the backplane via thecontrol and timing module 30. The server units 22 allow for additionalcompute capacity to be provided to the system in a modular fashion andallow for independent software delivery. The server unit 22 can beutilized to implement Virtualized Network Functions (VNFs), such as, forexample, Route Reflectors; Mapping Servers; OTN control plane for legacyinterworking; Telemetry monitoring, aggregation, and analytics;Localized VNF, example: Nodal Health and Service Assurance; and thelike.

The hardware platform 10 includes a backplane 36 that includes avariable size passive metal cassette 12 that provides connectivitybetween the horizontal units 14, 16, 18, 20, 22. In an embodiment, thebackplane 36 and the cassettes 12 utilize Twinaxial cabling (alsoreferred to as “Twinax”). Twinax is a type of cable similar to coaxialcable, but with two inner conductors instead of one. Due to costefficiency, it is becoming common in very-short-range high-speeddifferential signaling applications. Twinax has 0.2 dB/inch insertionloss relative to 1.0 dB/in for a reasonably high-end PCB material likeMeg7N+. So Twinax can have lengths 5× longer than non-exotic PCBmaterial, which enables larger system sizes.

FIG. 2 is a block diagram of a front view of an example hardwareplatform 10A supporting 230 Tb/s in a single rack. The hardware platform10A is presented for illustration purposes, and those of ordinary skillin the art will recognize various different configurations are possiblegiven the modular nature of the hardware platform 10. FIG. 2 is a frontview illustrating rack posts 40 and the associated units 14, 16, 18, 20.In this example, there are 8 packet units 14 with 8×36 QSFPs and 8universal units 16 with 8×12 USSMs 26. These units 14, 16 areinterconnected by the 6 fabric units 18 equipped with 12 switch modules28. For example, the 8 packet units 14 can provide local clientinterconnect while the 8 universal units 16 can provide remote clientinterconnect via DWDM.

FIGS. 3A-3D are diagrams of a single rack hardware platform 10B whereFIG. 3A is a side view, FIG. 3B is a front view, FIG. 3C is a rearsection view, and FIG. 3D is a rear view. In this example, the hardwareplatform 10B includes 28 packet units 14 that are interconnected to oneanother via 4 fabric units 18. Those of ordinary skill in the artrecognize a typical telecom rack can have four posts 40. In FIG. 3A, theside view illustrates the two posts 40 on one side and, in FIGS. 3B-3D,the front and rear views illustrate the posts 40 on opposite sides. InFIG. 3A, it is noted the units 14, 18, 20 take about half the depth ofthe rack. The cassette 12 can have a variable depth depending on howmuch interconnect is necessary, i.e., the Twinax cabling expands basedon how much interconnectivity is required. For example, the followingprovides some examples of the overall depth of the hardware platform 10based on the size of the system and the corresponding size of thecassettes 12—

Extra-Large systems (400 Tb/s) 900 mm Large system (100 Tb/s-200 Tb/s)800 mm Medium system (28.8 Tb/s-100 Tb/s) 700 mm Small, fabric-lesssystem (14.4 Tb/s) 600 mm

For example, a small fabric-less system may include a single packet unit14 with no corresponding cassette 12. The cassettes 12 includeconnectors 42 that mate mechanically with connectors 44 associated withthe units 14, 16, 18, 20. The cassettes 12 can include connectors 46that mate with connectors 48 on the fan modules 34.

FIG. 3C illustrates a rear sectional view of the hardware platform 10Bwith power modules 24 (PM) installed on the packet units 14 and thecontrol unit 20. There are no power modules 24 on the fabric units 18.The cassettes 12 are built to order as described herein. FIG. 3Dillustrates a rear view illustrating the 2 RU fan modules 34.

FIG. 3E is a block diagram of a hardware platform 10C in a side view tocompare a cassette backplane 12 with a backplane-less embodiment. Here,a smaller system with the packet units 14 and the fabric units 18 can beimplemented without the cassettes 12 to fit within 600 mm depth. In lieuof the cassettes 12, the hardware platform 10C utilizes faceplate AOCcabling.

Platform Software Features

The hardware platform 10 supports a full suite of routingprotocols/services expected of a next-generation transit core IPplatform—including feature-rich support for ISIS, BGP, and SegmentRouting. Also, as the trend continues towards extendingcontent/peering/compute capacity closer to the user, it is expected thatthe Service Edge infrastructure will remain the focal point for highdensity edge services and peering functions, allowing the core to beoptimized for transport scale/capacity. While it is assumed that thecore routers will not replace existing edge routers (e.g., MSE, PeeringRouter, etc.), the hardware platform 10 also supports L2/L3 services forlocations where a service provider could optimize footprint. Thoughinter-working may be required between the legacy core and the newnetwork, BGP, ISIS, and Segment Routing will form the maininfrastructure elements of the network, and legacy interoperability isperformed on a case-by-case basis. Particularly, where legacy OTNservices are required, the hardware platform 10 implementsOTN/Ethernet-emulation over its packet forwarding infrastructure (usinga specific USSM 26 variant). This will also provide service providersthe flexibility to carry legacy service types over their next-generationnetwork.

Platform Building Block Re-Use

Most of the active components associated with the hardware platform 10are shared across markets and applications. This allows the high-volumeaccess market to drive lower cost for the code market with the followingshared components: the fabric units 18, the switch modules 28, the powermodules 24, the control and timing module 30, and the fan modules 34.That is, the reuse of these common modules allows cost savings/sharing.

The power modules 24 have dual input feeds to support an A+B input feedscenario and an N+1 load sharing of power modules onto a single internalC-feed to minimize the feed requirements that are then distributed onthe virtual backplane to all components of the hardware platform 10.There can be separate modules for AC and DC. The fan modules 34 can havedual-rotors that provide higher fault tolerance with redundant powerfeeds and independent controls. Fan speed control is a closed-loopsystem with ambient temperature being utilized to determine the maximumfan speed and internal component temperatures utilized to slow down fansto maintain optimal component temperatures and minimize unnecessarypower consumption and noise.

Variable Fabric Size

FIG. 4 is a block diagram of a front view of a single rack illustratinghardware platforms 10-1, 10-2, 10-3 of varying sizes. An aspect of theapproach for the hardware platform 10 is that each system size has thenumber of fabric units 16 that it needs (e.g., 3, 4, or 12 switchmodules 28 per system). An example of the variable size is shown in FIG.4. There is no minimum number of fabric units 16. There can be zerofabric units 16 to realize a very cost-effective and compact direct meshat 14.4 Tb/s. Within a single market or application, each system size isdifferentiated only by the passive components. That makes it easier tocreate more unique sizes to fit the service provider's needs withoutoversizing any given node.

Multi-Rack Configuration

FIG. 5 is a block diagram of a front view of three racks illustrating amulti-rack configuration for a hardware platform 10D. In the past,back-to-back chassis used optical interconnect, which had two issues,namely high cost and requirement for new active fabrics to be developedand tested. The hardware platform 10D achieves a single multi-rack nodevia an unfolded ortho structure that has vertical fabrics in a centerrack 50 and horizontal interface units 14, 16, 18, 20 in two adjacentracks 52, 54. Of note, the center rack 50 includes the fabric units 18in a vertical orientation whereas the cassettes 12 are in a horizontalorientation.

In the example of FIG. 5, the hardware platform 10D includes aninterface rack 52 supporting 403 Tb/s, e.g., 20× packet units 14 and 8×universal units 16, and an interface rack 54 with 12× universal units16. The fabric rack 50 includes 9 fabric units 18 and 14 cassettes 12.The cassettes include connections 56 between the racks 50, 52, 54.

The connectivity is via passive Twinax cassettes each with 384 Twinaxlinks (38.4 Tbps). Due to the low insertion loss of Twinax, no repeatersor addition fabric devices are required. Compared with AOC cables, theTwinax cassettes 12 are roughly 10× less expensive. The cassettes 12also mate many more links simultaneously and thus reduce operationalinstallation and complexity relative to AOC cables.

Scaling

A hardware platform 10 instance has massive scale flexibility. Startingat 14.4 Tb/s, the hardware platform 10 can scale beyond 800 Tb/s+,through the addition of the appropriate number of unit 14, 16, 18instances. A unique property of this approach is that each system sizeis given the number of fabric units 18 that it needs (e.g., 3, 4, or 12switch modules per system). There is no minimum number of fabric units18. One can go all the way down to zero fabric units 18 and realize avery cost-effective and compact direct-mesh 14.4 T fabric-less smallsystem. Or have a greater number of fabric units 18 for larger systemseven extending to multiple racks. Within a single market or application,each system size is differentiated only by the passive components. Thissimplifies the overall design, enhances Service provider's Total Cost ofOwnership and reduces time to market.

The hardware platform can scale in 3 different dimensions, namelyinterface capacity, fabric capacity, and control plane capacity.

For interface capacity, adding additional packet unit 14 or universalunit 16 instances to a frame, additional interface/forwarding capacityis added. The packet units 14 are 1 RU and can be based on 2×Jericho3,36×QSFP400 faceplate (14.4 T), x86+FPGA accelerator for local control,and OAM. Adding additional packet unit 14 or universal unit 16 instancesprovides the addition of interface/forwarding capacity in increments of14.4 Tb/s×28 (28 packet unit 14 instances would be supported in a singlerack) to 403 Tb/s initially and expandable to 800+Tb/s with a multi-racksolution. The resulting port capacity would be:

Port Type Single Rack Platform Multi-Rack Platform 100 GE ports 40328064 200 GE ports 2018 4032 400 GE ports 1008 2016

For fabric capacity, adding additional fabric unit 18 instances to aframe, the overall switching capacity of the system is increased. Thefabric units 18 (1.25 RU) can be based on 2×Ramon3. Connecting thefabric unit 18 instances within a rack is done through a set of passive,Twinax-based cassettes 12 which reside behind the fabric unit 18. Twinaxhas 0.2 dB/inch insertion loss relative to 1.0 dB/in for a reasonablyhigh-end PCB material like Meg7N+. So Twinax can have lengths 5× longerthan non-exotic PCB material, which enables larger system sizes. Note:the larger the system, the deeper the backplane becomes.

For control plane capacity, the control plane capacity of a hardwareplatform 10 can be increased in-service by replacing the current controlunit 20 instance with a higher capacity instance. This can utilize anx86 on the control and timing module 30 to enable the higher scalerequired in 800 T+ systems. Local shelf management can be via a 25GEout-of-band control plane. The server unit 22 can also be used tooffload functions.

The dynamic scaling of the hardware platform 10 is meant to allow forthe same capacity scalability as is offered by traditional “Leaf &Spine” architectures, but without the operational complexity associatedwith fiber management; the need to dedicate a significant amount ofleaf, and spine capacity associated with inter-system interconnect; andthe added cost of the optical modules associated with inter-systeminterconnect.

For multiple racks, the control units 20 in different racks can connectone another via a faceplate optical ring.

Decoupled Control and Data Plane

The control unit 20 is a distinct component of the hardware platform 10and is not built to reside within the system. The control unit 20 canexecute an external control plane. For larger systems, the control unit20 could be installed in a separate rack(s) from the fabric unit18/packet unit 14 elements. Unlike traditional switch/routers, thecontrol unit 20 design is not limited by the constraints of the overallchassis system. As it is completely decoupled, the control unit 20 canscale completely independently from the switching/forwarding elements inthe system. When/if additional control plane capacity is required, anupdated control unit 20 can be installed in the system.

Environmental and Power Design

The physical architecture of the hardware platform 10 can provide directfront-to-back airflow as the fans 34 are located at the rear of thesystem. The fans 34 are located behind the Twinax cassette 12 allowingtrue front-to-back airflow. In this design, the units 14, 16, 18, 20elements are placed individually in a rack, and not stacked one behindthe other. This eliminates scenarios where the air traversing the fabricunit 18 is pre-heated by the forward placed modules (e.g., as intraditional ortho-direct systems).

The hardware platform 10 uses front-to-back airflow, with fans coveringalmost the entire rear of the system. Since the fan size is decoupledfrom the unit 14, 16, 18, 20 slot pitches, larger fans can be employedthat lower acoustic noise levels relative to smaller fans. The 2 RUdual-rotor fan modules 34 offer a good balance between airflowefficiency, noise, and the ability to be a common module with smallsystems too. Compared with larger fans, the 2 RU fans also reduce thesize of the airflow short-circuit introduced when a fan module isreplaced.

The hardware platform 10 is front access for everything except power andfans, so units 14, 16, 18, 20 do not preheat each other. This enables agood thermal margin.

With front-to-back airflow, the fans 34 are in the rear, so some amountof rear access may be required. The hardware platform 10 also includesthe power modules 24 in the rear to simplify power cabling and tomaximize the faceplate area available for client interfaces. However,everything else (including the fabric units 18, packet units 14, controlmodules 20, etc.) are in the front.

In an embodiment, all of the units 14, 16, 18, 20, 22 of the hardwareplatform 10 are air-cooled with the rear-mounted fans 34. Airflow isdirect from front-to-back without “turning” the air. All of the units14, 16, 18, 20, 22 can have faceplate perforations for intake air. Theair passes through the front optical modules, midplane openings, andthen through rear-mounted fans. In an embodiment, the maximum heatdissipation is ˜2 kW per RU of rack space for switching gear. For largesystems, the use of a liquid chilled rear door may be used to transferheat generated by the air-cooled system to water for removal from thebuilding. The use of the chilled rear door reduces or eliminates theload on Air-Conditioning Units.

With newer networking gear being designed with airflow directlyfront-to-back and air intake through faceplate perforations aroundoptical module cages, air filters may be required to be placed in a dooror cover over the front of the chassis. The space between the front ofthe shelf and the door must be enclosed at the top and bottom of thesystem to prevent system intake air from bypassing the air filter in thedoor. On the left and right side, brushes may be used that allow thefiber cables to pass through behind the door, at the same time blockingair from passing.

Liquid cooling comes in two basic forms—1) cold plates with hoses withcooling liquid running through them or 2) immersion liquid. In the caseof cold plates with hoses, this technology is used in high-performancecomputers. The pumped fluid does not come into direct contact withelectronic or optical components and remains in the cooling loop. In theimmersion case, dielectric fluid is in direct contact with theelectronics and must be sealed from the optical components whileproviding heat transfer. Immersion cooling is used in data centercompute systems using a bath where the gear is submerged from the top.There are emerging technologies that allow immersion module-levelcooling for use in traditional rack systems. For use in carrier-gradenetworking gear, both solutions require fluid to be pumped with dual,replaceable high-quality pumps.

The introduction of liquid cooling into switching gear requiresacceptance by operators as it impacts the office environment where theequipment is deployed. CDUs (cooling distribution units) must bedeployed along with the installation of delivery and return lines toequipment racks. Some operators are already deploying these structuresfor use on cabinet rear chilled doors to neutralize the increase inexhaust air temperatures in high power air-cooled equipment. The use ofchilled rear doors solves the problem of keeping control of office airtemperatures but provides no direct benefit for the cooling of theequipment. The use of chilled rear doors provides an adequate solutionfor rack densities to 50 kW. Liquid cooling of the equipment moves theliquid delivery lines from the chilled rear door, directly to theequipment, further increasing cooling operation efficiency byeliminating air from the cooling equation.

In the case where the equipment becomes liquid-cooled, but the officedoes not supply liquid cooling to the rack, the solution is to useliquid to air exchangers that transfer the heat removed from the gear inliquid back to the room air. This solution requires more space, is muchless efficient than the CDU solution and does not manage room airtemperature.

The hardware platform 10 can include the inclusion of liquid coolinginto its next-generation switching gear products. The first use ofliquid cooling will likely be in the form of a hybrid system using coldplates and hoses for liquid cooling the higher power components whilestill using air for some percentage of lower power components in thesystem. The use of liquid cooling where the air is eliminated from thecooling design has the benefit of dramatically improving Power UsageEffectiveness (PUE) over an air-cooled system. The energy used tooperate a liquid cooling system is much lower than that of system fanpower combined with required air conditioning in the customer premises.There is also power savings and an increase in reliability of thenetworking gear by keeping the electronics at a constant and lowertemperature versus the ambient operating temperature. Liquid cooling isthe future of networking gear.

There are three main factors to motivate moving from forced air coolingto liquid cooling, namely

1) equipment vendor maximizing bandwidth density in less space;

2) operators not able to manage the temperature of the room due to rackpower density; and

3) Interest in improving operating efficiency—improving PUE.

To date, the physical size of network switching gear products has beendriven by the faceplate space required for the IO and the module spacerequired to house the electronics. Power density has allowed forced airto adequately cool these systems with the space available for heat sinksto transfer the heat from the electronics to the air. As systems havegrown in capacity, the Watts per Gigabit ratio continues to drop witheach technology node at the cost of higher power density. In the nextseveral years, power densities will be reached where the space availablefor heat sinks will not be adequate to transfer heat from theelectronics. The choice is to remain air-cooled with larger heatsinksand therefore, the size of the system will then be driven by air-cooledcapability or to move to more efficient means of cooling, liquidcooling, to continue to have required IO space drive the physical sizeof systems.

Ortho-Direct Chassis Approach

FIGS. 6A-6D are block diagrams of an ortho-direct approach for a chassis60, for comparison with the hardware platform 10. The chassis 60includes fabric modules 62 that are vertically oriented, to connect tothe horizontally oriented units 14, 16, 20. A fundamental issue withortho-direct fabric modules 62 is that they become unwieldy to installand replace as system size grows past a certain point. For example, theunwieldy point can be around 36″-tall (20 RU) for fabric modules 62.That height limit can accommodate 20 packet units 14 (1 RU and 14.4 Teach). So, the maximum system size is 288 Tb/s. For a packet-onlysystem, this might be reasonable in many applications.

However, if integrated lines are desired, this requires a mix of packetunits 14 and universal units 16, which reduces density due to the 20 RUheight limit imposed by the fabric modules 62. For example, with 5universal units 16 (3 RU and 14.4 T each) and 5 packet units 14 (1 RUand 14.4 T each), the maximum system size drops from 288 Tb/s to 144Tb/s (72 Tb/s client+72 Tb/s line).

Alternatively, the ortho-direct system can be populated only with packetunits 14 (20 total), and half of them can connect to external DWDMsystems via AOC 400 G cables. This results in 144 Tb/s client+144 Tb/sLine but requires the cost and fiber management of 360 AOC 400 G cablesalong with extra rack space for the external DWDM systems.

To break through the maximum system size imposed by unwieldy fabricmodules 62, multiple small fabric modules 62 can be joined with fabricmodule couplers 64, as illustrated in FIGS. 7A and 7B. This addsinstallation complexity, lowers expansion granularity, increases depth,and worsens signal integrity.

FIG. 8 is a table illustrating different approaches for networkinghardware to compare the hardware platform 10 with the chassis 60 andpizza boxes. As seen in this table, the hardware platform 10 achieves abetter set of trade-offs (relative to the chassis 60 and pizza boxes)that enables a larger maximum size, front access, easier cablemanagement, shallower depth, higher ambient temperature operation,allows different numbers of fabrics in different sizes, and uses thesame equipment across many sizes.

Of particular note, the hardware platform 10 is designed so that allunits 14, 16, 18, 20 occupy their own space vertically. That is, none ofthe units 14, 16, 18, 20 are parallel to one another in the verticaldirection, and thus, all airflow (either front-to-back or side-to-side)is efficient. The chassis 60, on the other hand, has the fabric modules62 behind the packet units. Here, the air cooling is such that thefabric modules 62 receive heated air, i.e., that has already flowed overthe packet units.

High Availability and Redundancy Options

Historically, traditional systems have tied the control plane, dataplane, and OAM (management) plane together so that they could not bescaled separately. The hardware platform 10 is designed with separatelyscalable planes in mind since each plane does not have to grow at thesame rate. FIG. 9 is a block diagram of a packet unit 14, a server unit22, and a control unit 20 for describing control plane and OAM planescalability. The packet unit 14 an x86 processor 70, an FPGA 72, and anEthernet switch circuit 74 (e.g., a Jericho3). The server unit 22 caninclude one or more x86 processors 70 and a Network Interface Card (NIC)76. The control unit 20 includes one or more x86 processors 70 and theaccess panel 32 connected to an Ethernet switch 78. Unitinterconnectivity can be via the backplane cassette 12 between thepacket unit 14 and the control unit 20, and via faceplate AOC betweeneach of the units 14, 22, 20.

In order to make each plane separately scalable, the hardware platform10 supports an N+1 load-sharing architecture. So the control units 20can be N+1 in the future as protocols develop that can be spread acrossmultiple CPUs. And x86 servers can be added to scale parts of the OAMplane, such as via the server unit 22.

Power Distribution

Again, the power density of networking hardware is continuouslyincreasing with bandwidth. Over the next several generations, largenetworking systems that take up a rack will grow in power from 20 kW perrack to 50-100 kW per rack. At these levels, it is not sensible to usetraditional 48 VDC office feeds to supply power to this high equipment.Overall, too much supply current is required, impacting the size andnumber of delivery feeds to the equipment as well as impacting the sizeof conductors, protection devices, and front-end power circuits withinthe system being powered. A 100,000 W system requires 2500 A of currentat 40 VDC. A standard has been developed to solve this problem using 380VDC (+/−190 VDC) which drops the current to 1/10 that required in 48 VDCsystems.

In order to improve the operational efficiency of power delivery, notonly does the input and system internal voltages used for power deliveryneed to be increased, a reduction in the number of power conversionsbetween the power grid and the equipment should be made. Every powerconversion is a penalty in terms of power to perform the conversion andthe monetary cost of purchasing the equipment. For example, conversionfrom 480 VAC three-phase directly to 380 VDC external to the equipmentwould be more efficient than an additional step to 240 VAC single phasefirst. 120 VAC just like 48 VDC, is not a practical power voltage forlarge systems. Delivering the highest voltage possible as close aspossible to the equipment inputs, reduces IR drop and the size of copperconductors used to carry the current. Inside the switching gear, 380 VDCis converted to 48V or 12V on the individual modules that is thenconverted to various voltages from 0.8V to 3.3V to directly power theelectronics.

In an embodiment, the hardware platform 10 uses substantially 380 VDC todistribute power within the system. Using N+1 load-sharing power modules24, the hardware platform 10 supports numerous office feed voltages withthe use of different power module 24 models including 48 VDC, 380 VDC,240 VDC/240 VAC, 480 VAC. The power modules 24 convert these officevoltages to 380 VDC for distribution to the loads in the hardwareplatform 10.

In an embodiment, the power modules 24 are N+1 protected (1 protectionpower module 24 for N working power modules 24, N is an integer greaterthan or equal to 1). Each power module 24 can include two input feeds(e.g., A and B feeds), and each power module 24 can perform load sharingto provide an output feed (e.g., a C feed). There can be two outputfeeds in each housing, e.g., a C1 feed for a working zone and a C2 feedfor a protection zone.

The power modules 24 can be inserted in the housing at any position andmate with the appropriate busbars and bused backplane. Multiple powermodules 24 can load share into one housing. The power modules 24 can bemounted outside the housing and attached directly to the rack.

The control unit 20 can detect that communications are lost to a powermodule 24 because it can see that the unit powered by that power module24 is up.

Switching Architecture

FIG. 10 is a diagram illustrating a leaf-spine architecture. FIG. 11 isa diagram comparing the hardware platform 10 to the leaf-spinearchitecture. There are tradeoffs between a switching architecture thatthat uses a single chassis versus distributed switching hardware (e.g.,leaf-spine). Compared to leaf-spine, typical chassis bandwidth islimited by the physical size of a single rack—either rack height ordepth or both. A leaf-spine architecture is naturally multi-rack capablesince it is interconnected with AOC cables. This means that only thecost of cabling, installation complexity of AOC cabling, and switchdevice radix are limited to its maximum system size. Growing beyond 800Tn/s+ should not be a problem with leaf-spine.

Leaf-spine pizza boxes have a better chance of being air-cooled at 50deg. C. since they can be made whatever height is necessary to achievethis. And since they are not limited to a single rack, the extra heightdoes not decrease maximum system size as it does with the othersingle-rack options.

In terms of comparing the hardware platform 10 to leaf-spine pizzaboxes, leaf-spine pizza boxes can more easily fit in 600 mm depth ifneeded (not including cabling room). However, leaf-spine pizza boxesconsume more rack units. To achieve an 800 Tb/s+ node built from 15nodes of 100 Tb/s each (see FIG. 10), this would consume 5.5 racks.Leaf-spine pizza boxes can be in a variable-pitch system, e.g., thatallows 1 RU, 2 RU, and 3 RU pizza boxes to be mixed together as part ofa single switch. Unlike fixed-slot-pitch chassis-based systems, a givenpizza box can be designed for whatever height is necessary for cooling,power, and type of client ports, whereas shelf-based systems requirefixed-pitch modules to match the fixed-pitch backplane.

One trade-off in a leaf-spine is that the faceplate is used for fabricconnectivity (via AOC's), so it reduces faceplate density available forclient ports. However, this reduction in density does not limit maxsystem size since the pizza boxes can sprawl across racks easily.Leaf-spine maximum size can be limited by switch radix, but this limithappens well above 800 Tb/s+. Also, if many leaf-spine pizza boxes arecontrolled via SDN, control plane scalability can become an issue.

The existential disadvantage of leaf-spine systems is the cost of thefabric interconnect via AOC's. For an 800 Tb/s+ non-blocking system,this can be 1250 individual AOC400 G cables or more for a non-blockingsystem constructed from fifteen 100 Tb/s chassis. If 25.6 Tb/s pizzaboxes were used, the number of cables increases to 3810 AOC's. The costof the AOC's can be reduced with Twinax QSFP400 DAC cables, but thisintroduces more stiffness, larger bend radius, and more bulk leading togreater difficulty in slack management.

At some point, system size is larger than chassis-based systems, andleaf-spine is the only way to scale. For these large systems, it can beadvantageous to build them up from larger hardware platform 10 nodes.This replaces a lot of the AOC cables with more cost-effective andoperationally simpler cassettes 12. It also improves control planescalability by reducing the number of unique management points in thenetwork.

As illustrated in FIGS. 10-11, although the leaf-spine architecturesolves many constraints required by chassis-based systems, the cablingcost and operational complexity are prohibitive for 800 Tb/s+non-blocking systems, unlike the hardware platform 10.

The dynamic scaling (in 3 different dimensions) of the hardware platform10 allows for the same capacity scalability as is offered by theleaf-spin architectures, but without the operational complexityassociated with fiber management; the need to dedicate a significantamount of leaf, and spine capacity associated with inter-systeminterconnect; and the added cost of the optical modules associated withinter-system interconnect.

Interface Support

The hardware platform 10 supports various interfaces, e.g., pluggablemodules. For example, the packet unit 14 can support 36×QSFPDD/QSFP28interfaces providing support for 100GE, 200GE, and 400GE clientinterfaces with the appropriate Ethernet FEC via QSFP28 QSFP56, QSFP-DD,etc. Additional client support can be via a universal unit 16 utilizingup to 12 USSMs 26 where each module is purpose-built for enhancedcapabilities such as FlexO (FOIC), OTU4, 10GE/40GE.

Hardware Platform Scale

FIG. 12 is a diagram of a hardware platform 10E illustrating multi-rackexpansion. Again, the hardware platform 10 is multi-rack back-to-backcapable, so scale is not limited by a single rack. This back-to-backapproach in the hardware platform 10E uses Twinax so it is lessexpensive than optical interconnects. It also uses passive modules, sono extra cost for repeaters. Whether in a single rack or two racks, thehardware platform 10E effectively operates as a single switching nodemanaged by the same control unit 20. That is, regardless of the numberof units 14, 16, 18, 20, 22 and the number of racks, the hardwareplatform 10 is a single network element from a control plane and networkmanagement perspective.

Fabric and Network Circuitry

In an embodiment, the hardware platform 10 can utilize the Broadcom DNXfamily of hardware to implement data path traffic forwarding andmanagement and uses the Fabric Access Processor ASIC (FAP) to performall packet processing functions and a Fabric Element (FE) ASIC toconnect all packet unit 10 elements together. The Fabric AccessProcessor (FAP) provides the following functions:

1) Flexible packet classification using a programmable microcode enginewith built-in support for data center and carrier networkingapplications. The system contains large on-chip classification databaseswhich can be extended using an external knowledge-based processor.

2) Traffic manager with deep packet buffers. The implementationdistributed scheduling scheme that allows state-of-the-art hierarchicalquality-of-service (QoS), transmission scheduling per-customer,per-service, as well as tunneling and overlay networks. Flexible flowcontrol mechanisms support Priority-based Flow Control (PFC), EnhancedTransmission Selection (ETS), Explicit Congestion Notification (ECN).

The FE provides the following capabilities:

1) Switching function between the FAP and provides a complete solutionfor switching fabric, traffic management, packet processing, and networkinterfaces.

2) Scalability is provided in three dimensions, which allows the user toconnect mixed-rate line cards: Port rates, Port counts, and Trafficmanagement services.

3) A self-routing device with dynamic load balancing.

4) Supports fabric multicast and multicast load balancing.

5) Supports three pipes for traffic allocation of Unicast Data—MulticastData and priority.

6) The fabric is a cell-based switch.

7) Automatic fault detection.

8) Scheduled device removal and device insertion without cell loss.

9) The system provides fault tolerance by automatically detecting andrerouting around faulty links or elements, resulting in a fault-tolerantself-healing fabric. The system reacts to faults in a device,connection, or configuration.

Packet and OTN Switching Support

The hardware platform 10 supports the capability to transparently mapOTN and Ethernet interfaces over a packet service using specializedUSSM's 26 within the universal unit 16, for transparent private lineservices. The USSM's 26 can provide an emulated service that is a packetservice providing either bandwidth reservation and admission control orforwarding prioritization and boundary traffic conditioning mechanisms.The network must provide guarantees for low jitter and low loss.

Long Haul Optics

The hardware platform 10 supports the direct integration ofhigh-performance long-haul optics (90+ GBaud) via an external shelf orvia the universal unit 16 and USSM's 26.

Cassettes and Hardware Implementation

FIG. 13 is a diagram of a rear view and a side view of the cassettes 12.FIG. 14 is a side view of the hardware platform 10 illustrating airflowbetween the units 14, 16, 18, 20, the cassettes 12, and the fans 34.FIG. 15 is a perspective diagram of an implementation of the cassettes12 and housings for the packet units 14 and the fabric units 18. FIG. 16is a front perspective diagram of the housings connected to thecassettes and mounted in a rack. FIG. 17 is a rear perspective diagramof the same hardware platform as FIG. 16. FIG. 18 is a horizontalsection view of the same hardware platform as FIG. 17. FIG. 19 is a rearview diagram of the housings illustrating the connectors to thecassettes 12.

The cassettes 12 can include a substantially rectangular housing thatextends vertically and connects horizontally to the various units 14,16, 18, 20, 22. In an embodiment, cables 80 inside the cassettes 12 canTwinax copper cables. As illustrated in FIG. 13, the depth of thecassettes 12 are based on how many units 14, 16, 18, 20, 22 aresupported vertically. In the side view of FIG. 13, the depth is shownexpanding as the number of units increases. This is due to the fact eachunit has to be connected to one another via a set of Twinax cables. Inan embodiment, the depth is about equal to

$\begin{matrix}{= {\left\lbrack {{{1/4^{''}}\mspace{14mu}{cable}} + {{1/4^{''}}\mspace{14mu}{space}}} \right\rbrack \times \left\lbrack {{Number}\mspace{14mu}{of}\mspace{14mu}{units}} \right\rbrack}} \\{= {8^{''}\mspace{11mu}{for}\mspace{14mu} 16 \times {units}}} \\{= {16^{''}\mspace{14mu}{for}\mspace{14mu} 32 \times {units}}}\end{matrix}$

Also, the cables 80 can span between adjacent racks to create a singlemulti-rack switch. The cables 80 delivers data fabric and controlsignals and can mate to power buses.

In another embodiment, the cables 80 can be optical fiber with MPO/MTPconnectors.

For example, in FIG. 18, the sectional view illustrates four cassettes12, each forming a channel that terminates with the fans 34. The evendistribution of cables 80 in the cassette 12 can ensure airflow isconfined in each channel.

Of note, the cables 80 are point-to-point connections. That is, each ofthe units on the right side of FIG. 13 has a corresponding cable 80connected to each other unit. As such, the more units, the larger therequired depth for the cassettes.

FIG. 19 illustrates the back of the housings, where connectors are shownfor each unit. The cassettes 12 span vertically and the units 14, 16,18, 20, 22 connect horizontally to the connectors. For example, in anembodiment, there are four cassettes 12, each having the cables 80, andeach spanning the vertical length of a rack. Each unit connects tocorresponding connectors on each of the four cassettes 12.

The cassettes 12 mate directly with the units, but also mate directly tothe housing for the purpose of reducing the number of physical dropsrequired of the cables 80.

The endpoints of the cassettes 12 (FIG. 19) provide ElectromagneticInterference (EMI) containment of a group of signals. The endpoints canprovide mounting, gasketing, power, and control signals for fan modules.

The cassettes 12 can be passive devices generally but may include anIDROM (ID Read Only Memory) for identification purposes.

The housing for the cassettes can include liquid-cooling manifolds andtubing for multiple units.

As described herein, the cassettes 12 include different depthscorresponding to different system sizes. The cassettes 12 can bedifferent based on the size/pitch of the units.

Networking Hardware Platform

Referring back to FIGS. 1-19, the present disclosure includes thenetworking hardware platform 10 which includes a plurality of units 14,18, 20 each having a first plurality of connectors 44 on a rear side,wherein each of the plurality of units 14, 18, 20 is any of a packetunit 14, a fabric unit 18, and a control unit 20, and whereincollectively the plurality of units 14, 18, 20 are configured to operateas a network element in a network; and a cassette 12 system includingone or more units (cassettes 12) each having cabling 80 therein and asecond plurality of connectors 42 located along the length to connect tocorresponding connectors of the first plurality of connectors 44,wherein the cassette 12 system is configured to connect to a rack (posts40), wherein an orientation of the one or more units is orthogonal to anorientation of the plurality of units, in the rack, wherein the cabling80, in each cassette 12, extends between each of the second plurality ofconnectors 42 for interconnectivity between each corresponding connectoralong the length of the corresponding cassette 12 (see FIG. 13).

The cables 80 can include a plurality of Twinaxial cables. Thenetworking hardware platform 10 can further include connectors 102associated with the cassette 12 system configured to connect tocorresponding connectors in a second rack, such that the network elementincludes associated units in the rack and in the second rack. Thenetworking hardware platform 10 can further include a second controlunit 20 located in the second rack connected to a first control unit 20located in the rack. The cables 80 include optical cabling with amulti-fiber connector forming the first plurality of connectors 44 andthe second plurality of connectors 42. The networking hardware platform10 can further include one or more fans 34 disposed at an opposite endof the one or more units 12 from the plurality of units 14, 18, 20. Theone or more units 12 can be multiple units that form channels forairflow between the plurality of units 14, 18, 20, and the fans 34 andfor electromagnetic interference (EMI) containment. Each of theplurality of units 14, 18, 20 can be in a different horizontal planefrom one another and airflow is front-to-back from a faceplate of eachhorizontally oriented unit 14, 18, 20 to corresponding one or more fans34.

The cassette 12 can be a passive device and includes a read-only memoryfor identification thereof. A depth, horizontally, of the one or moreunits 12 can be based on a number of the plurality of units 14, 18, 20.A pitch of at least two units of the plurality of units 14, 18, 20 isdifferent. The packet units 14 can be configured to support pluggableoptical modules. The networking hardware platform can further include atleast one universal unit 16 that is similar functionally to the packetunit 14 except for larger pitch and for support of at least one coherentoptical module for long haul transmission. The networking hardwareplatform can further include N+1 power modules 24 located at a rear ofthe cassettes 12, N is an integer ≥1, and the +1 is a protection module.The N+1 power modules 24 can provide substantially 380 VDC to the units.

FIG. 20 is a flowchart of a process 120 of installing a networkinghardware platform 10. The process 120 includes:

installing a cassette system in a rack, wherein the cassette systemincludes one or more units each having cabling therein and a firstplurality of connectors located along the length, wherein the cabling,in each unit of the cassette system, extends between each of the firstplurality of connectors for interconnectivity between each correspondingconnector along the length of the corresponding unit of the cassettesystem (step 122); and

connecting a plurality of units to the cassette system, wherein each ofthe plurality of units has a second plurality of connectors on a rearside configured to connect to corresponding connectors of the firstplurality of connectors, wherein an orientation of the plurality ofunits is orthogonal to an orientation of the one or more units, whereineach of the plurality of units is any of a packet unit, a fabric unit,and a control unit, and wherein collectively the plurality of units areconfigured to operate as a network element in a network (step 124).

The cables can include a plurality of Twinaxial cables. The process 120can further include connecting the first plurality of connectorsassociated with the cassette system to corresponding connectors in asecond rack, such that the network element includes associated units inthe rack and in the second rack.

It will be appreciated that some embodiments described herein mayinclude or utilize one or more generic or specialized processors (“oneor more processors”) such as microprocessors; Central Processing Units(CPUs); Digital Signal Processors (DSPs): customized processors such asNetwork Processors (NPs) or Network Processing Units (NPUs), GraphicsProcessing Units (GPUs), or the like; Field-Programmable Gate Arrays(FPGAs); and the like along with unique stored program instructions(including both software and firmware) for control thereof to implement,in conjunction with certain non-processor circuits, some, most, or allof the functions of the methods and/or systems described herein.Alternatively, some or all functions may be implemented by a statemachine that has no stored program instructions, or in one or moreApplication-Specific Integrated Circuits (ASICs), in which each functionor some combinations of certain of the functions are implemented ascustom logic or circuitry. Of course, a combination of theaforementioned approaches may be used. For some of the embodimentsdescribed herein, a corresponding device in hardware and optionally withsoftware, firmware, and a combination thereof can be referred to as“circuitry configured to,” “logic configured to,” etc. perform a set ofoperations, steps, methods, processes, algorithms, functions,techniques, etc. on digital and/or analog signals as described hereinfor the various embodiments.

Moreover, some embodiments may include a non-transitorycomputer-readable medium having instructions stored thereon forprogramming a computer, server, appliance, device, processor, circuit,etc. to perform functions as described and claimed herein. Examples ofsuch non-transitory computer-readable medium include, but are notlimited to, a hard disk, an optical storage device, a magnetic storagedevice, a Read-Only Memory (ROM), a Programmable ROM (PROM), an ErasablePROM (EPROM), an Electrically EPROM (EEPROM), Flash memory, and thelike. When stored in the non-transitory computer-readable medium,software can include instructions executable by a processor or device(e.g., any type of programmable circuitry or logic) that, in response tosuch execution, cause a processor or the device to perform a set ofoperations, steps, methods, processes, algorithms, functions,techniques, etc. as described herein for the various embodiments.

Although the present disclosure has been illustrated and describedherein with reference to preferred embodiments and specific examplesthereof, it will be readily apparent to those of ordinary skill in theart that other embodiments and examples may perform similar functionsand/or achieve like results. All such equivalent embodiments andexamples are within the spirit and scope of the present disclosure, arecontemplated thereby, and are intended to be covered by the followingclaims.

What is claimed is:
 1. A networking hardware platform comprising: aplurality of units each having a first plurality of connectors on a rearside, wherein each of the plurality of units is any of a packet unit, afabric unit, and a control unit, and wherein collectively the firstplurality of units are configured to operate as a network element in anetwork; and a cassette system including a one or more units each havingcabling therein and a second plurality of connectors located along thelength to connect to corresponding connectors of the first plurality ofconnectors, wherein the cassette system is configured to connect to arack, and wherein an orientation of the one or more units is orthogonalto an orientation of the plurality of units, in the rack, wherein thecabling, in each of the one or more units, extends between each of thesecond plurality of connectors for interconnectivity with correspondingconnector along the length of the corresponding one or more units. 2.The networking hardware platform of claim 1, wherein the cablingincludes a plurality of Twinaxial cables.
 3. The networking hardwareplatform of claim 1, wherein the second plurality of connectors arefurther configured to connect to corresponding connectors in a secondrack, such that the network element includes the plurality of units inthe rack and in the second rack.
 4. The networking hardware platform ofclaim 3, further comprising: a second control unit located in the secondrack connected to a first control unit located in the rack.
 5. Thenetworking hardware platform of claim 1, wherein the cabling includesoptical cabling with a multi-fiber connector forming the first pluralityof connectors and the second plurality of connectors.
 6. The networkinghardware platform of claim 1, further comprising one or more fansdisposed at an opposite end of the one or more units from the pluralityof units.
 7. The networking hardware platform of claim 6, wherein theone or more units include multiple units that form channels for airflowbetween the plurality of units and the fans and for electromagneticinterference (EMI) containment.
 8. The networking hardware platform ofclaim 6, wherein each of the plurality of units are in a different planefrom one another and airflow is front-to-back from a faceplate of eachunit to corresponding one or more fans.
 9. The networking hardwareplatform of claim 1, wherein the one or more units are each a passivedevice and includes a read-only memory for identification thereof. 10.The networking hardware platform of claim 1, wherein a depth of the oneor more units is based on how much interconnect is required for theplurality of units.
 11. The networking hardware platform of claim 1,wherein a pitch of at least two units of the plurality of units isdifferent.
 12. The networking hardware platform of claim 1, wherein thepacket units are configured to support pluggable optical modules. 13.The networking hardware platform of claim 1, further comprising at leastone universal unit that is similar functionally to the packet unitexcept for larger pitch and for support of at least one coherent opticalmodule for long haul transmission.
 14. The networking hardware platformof claim 1, further comprising N+1 power modules located at a rear ofthe one or more units, N is an integer ≥1 and the +1 is a protectionmodule.
 15. The networking hardware platform of claim 14, wherein theN+1 power modules provide substantially 380 VDC to the units.
 16. Amethod of installing a networking hardware platform comprising:installing a cassette system in a rack, wherein the cassette systemincludes one or more units each having cabling therein and a firstplurality of connectors located along the length, wherein the cabling,in each unit of the cassette system, extends between each of the firstplurality of connectors for interconnectivity between each correspondingconnector along the length of the corresponding unit of the cassettesystem; and connecting a plurality of units to the cassette system,wherein each of the plurality of units has a second plurality ofconnectors on a rear side configured to connect to correspondingconnectors of the first plurality of connectors, wherein an orientationof the plurality of units is orthogonal to an orientation of the one ormore units, wherein each of the plurality of units is any of a packetunit, a fabric unit, and a control unit, and wherein collectively theplurality of units are configured to operate as a network element in anetwork.
 17. The method of claim 16, wherein the cabling includes aplurality of Twinaxial cables.
 18. The method of claim 16, furthercomprising: connecting the first plurality of connectors associated withthe cassette system to corresponding connectors in a second rack, suchthat the network element includes the plurality of units in the rack andin the second rack.
 19. A networking hardware interconnection systemcomprising: one or more units that are configured to connect to a rack;and cabling in each of the one or more units between a first pluralityof connectors located along the length, wherein the first plurality ofconnectors are configured to connect to corresponding connectors of asecond plurality of connectors of a plurality of units that areorientated orthogonal to the one or more units, and wherein the one ormore units form channels for airflow between the plurality of units andfans and for electromagnetic interference (EMI) containment.
 20. Thenetworking hardware interconnection system of claim 19, wherein thecabling includes a plurality of Twinaxial cables.